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Abstract 

Counseling psychology is a burgeoning field still in its inchoate stages, attempting to 
gain/maintain autonomy and respect. As students of a scientific-practicing counseling 
psychology program, it behooves us to conduct well thought-out, meaningful research in 
the name of practicing “good science,” as does it benefit all counseling psychologists in 
the name of furthering the field’s namesake. Unfortunately, many times the tendency to 
embark on a research endeavor lacks the necessary foresight in constructing the design. 
Research designs are pervious to many different types of threats to their internal and 
external validity. In the traditions of Campbell and Stanley, and Cook and Campbell, this 
paper will elucidate some of the more common types of research designs, along with the 
coexistent threats to validity. Further, an example of each type of design has been given 
from the counseling literature for the interested reader to paruse and help make the 
concepts concrete. 
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A poem by Schuyler W. Huck (1991): 

True Experimental Design 
R OiX O2 
R 0 \ O2 

There once was a research design 
That looked, on first glance, oh so fine 
Yet it is stupidity 
To view its validity 
As chaste due to Rs on each line 

For one thing, mere randomization 
That’s done to gain equalization 
Of Es versus Cs 
Gives no guarantees 
That you’ll have good gen’ralization 

Results may, for instance, show all 
That Es topped the Cs (with p small) 

But if that full troop 
Aint your “target” group 
Wide claims will lead to your downfall 

Or X may work well for just some 
While others to it are quite numb 
The treatment’s true fate 
Depends on the trait 
One needs to gain from X shall come 

The one who rates subjects may know 
Which S’s got treatment-and so 
Effect Rosenthal 
May ruin it all 

And truth from your study won’t flow 

If treatment is novel, each E 
May first say “It’s super for me!” 

But as time goes by 
E may think and sigh 
“No value in X do I see” 
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Or just the reverse can occur 
With X thought at first to deter 
But once acclimated 
E’s may be elated 
And to X high rank they’ll confer 



Should hist’ry and X interact 
‘Tis true there is no artifact 
But if replicated 
You’ll be decimated 
By findings that prove inexact 



The rV or DV may be 
Reported not sufficiently 
If later one tries to 
Your study to redo 
New findings may be what you’ll see 

If two or more studies are done 
With S’s not used in the just one 
A treatment... it’s “grade” 

May strengthen or fade 
If only one study is run 

A pretest or posttest may make 
The E group to X wide awake 
O leads to creation 
Of sensitization 

And sans O the treatment won’t take 

The time from the X to O2 
Is worthy of careful review 
For if it’s revised 
When new plans devised 
Fresh findings may seem quite askew 

The subjects may be told or know 
Observers are watching... and so 
They’ll work with great skill 
Only until 

The watchers stop watching and go 
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Or what if the subjects perceive 
What you hope and want and believe 
Unconscious or not 
Like robots they trot... 

And data you want you’ll receive 

We hate now to sound so paternal 
But we must move past threats “external” 

More pitfalls there are 
To rip, maim, and scar 
And keep your work out of a journal 

Suppose X or the O thought 
To deal with a construct that’s “caught” 
When such is not true 
With “fit” that’s askew 
One’s main claims won’t be what they ought 



Moreover, one should not assume 
That stat work can’t add to the gloom 
Assumptions may be 
False prima facie 

And that makes the truth not illume 



Or what if theres meager precision? 

Strong Xs you’ll miss for poor “vision” 

The Error Type II 
Will hide truth fi'om view 
And you’ll make an inapt decision 

But what of those “internal” threats? 

Are all tamed, like nicely trained pets? 

If that rings true 
You’ve learning to do 
For R’s catch just some in their nets 

For instance, take treatment diffusion 
Which can make lots of confiision 
Should E give to C 
All X that there be 
You’ll come to a faulty conclusion 
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Or what if the folks in the C group 
Feel they were assigned to the “B” group 
They’re demoralized 
And you victimized 
By “findings” as murky as pea soup 

The converse may also take place 
C’s saying, “We will win the race!” 

Each tries like a kid 
(John Henry once did!) 

Yet this can one’s findings debase 

And what if there’s subject attrition 
Related to the X condition? 

It may just appear 
That X was “in gear” 

When that’s just a sad apparition 

Of course there is further the chance 
That someone may try to enhance 
The plight of those in 
The C group wherein 
Such help is a foul circumstance 

And lastly consider the stew 
You’re in if on post E’s construe 
On DV their standing 
As not so commanding 
Cause X brought respect for O2 

By “lastly” we sure don’t intend 
To mean that one need not attend 
To problems that have now 
No label, yet somehow 
They could wreck one’s work in the end 

The moral, one hopes, is quite clear 
The word “true” does not make it appear 
That problems of import 
Are all on the backcourt 
When really they may lurk quite near! ! ! 
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A Primer on Experimental and Quasi-Experimental Design 

Upon its inception as an experimental science, psychology has utilized the scientific 
method found in the physical sciences. In 1879 Wilhelm Wundt opened the first 
psychological laboratory in Leipzig, Germany, and with that commencement also came the 
first arguments about the validity of Wundt’s experiments (Benjamin, 1988). Since that 
time, the scientific method has been applied to various psychological constructs, e.g., 
behaviorism, learning theory. Gestalt psychology, animal experimentation, cognition, and 
functionalism (Gannon, 1991). 

Counseling Psychology has experienced many “growing pains” in its attempts at 
being recognized as a separate (Hiebert, Simpson, & Uhlemann (1992); see also Wooley, 
Duffy, & Dawson, (1996) for a preliminary study to support counseling psychology’s 
attempts at autonomy), viable science, which has been linked previously by some to the 
utilization of the scientific method (Hill & Corbett, 1993; Schultz, 1972). Since the 
inaguaration of counseling psychology in 1946 (Whitley, 1984) it was mainly an applied 
psychology. At this juncture in time, the application of psychology was only beginning to 
gain respect from the intact group of psychologists who considered themselves as “pure”— 
that is they engaged in experimental psychology (Benjamin, 1988). In light of this 
Zeitgeist and the identity struggles within the division, it stands to reason that counseling 
psychology places scientific inquiry through the rigor of the scientific method, as a core 
function. 

In the past 20 years, there has been growing dissension in the ranks of counseling 
psychology researchers regarding the way in which research focusing on the philosophy of 
science and counseling psychology is being conducted. Many believe that researchers are 
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placing too much emphasis on objectives and questions to which research should be 
directed, with little attention to actual research designs and methods (Ford, 1984; 
Goldman, 1976; Howard, 1982; Parker, 1993; Serlin, 1987; Serlin & Lapsley, 1985). 
Others have directly stated that more attention should be placed on the training aspects of 
research methodology (Gelso, 1979a, 1979b; Goldfried, 1984; Magoon & Holland, 1984), 
(note though that Birk and Brooks, 1986, report that 81% of 300 counseling 
psychologists surveyed reported adequate training in research). Indeed, the results of a 
national survey of counseling psychologists indicate that 49.6% of their time is devoted to 
the activity of research (Watkins, Lopez, Campbell, & Himmell, 1986), thus further 
supporting the relevance of the present paper to counseling psychologists. 

If this is the case, how is it that so many counseling psychologists and counseling 
psychology students are producing “bad science”? For example, in a specific instance of 
reviewing one (psychology-type) department’s recent dissertations, Thompson (1994) 
found numerous blatant mistakes due to methodological errors, and many others have 
challenged the current state of research (e.g., O’Hear & MacDonald, 1995). These errors 
are most likely representative and indicative of the more common mistakes found in 
research presently. 

With this backdrop and apparent need for remediation, the present paper presents 
a “primer on experimental designs,” with the specific goal of (review), and the more 
comprehensive intention that through “better and improved science”, counseling 
psychology will continue to solidify its place as an indisputably separate and viable field. 

In the traditions of Campbell and Stanley (1963) and Cook and Campbell (1979), a review 
of experimental and quasi-experimental designs and how threats to validity impacts 
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counseling research will be presented, employing examples from the current counseling 
literature. 

The Validity of Experimental Designs 

Internal validity 

Internal validity is one important type of research validity. The term “internal 
validity” refers to the extent that extraneous variables (error variance) in an experiment are 
accounted for. It is paramount to the researcher that model specification error variance 
(as distinct from measurement and sampling error variance) is controlled because if not, 
the researcher can not emphatically conclude that the observed outcome is due to the 
independent variable(s) (Parker, 1993). Campbell and Stanley (1963) stated that “internal 
validity is the basic minimum without which any experiment is uninterpretable” (p. 5). 
There are eight major threats to internal validity; (a) history, encompassing the 
environmental events occurring between the first and second observations in addition to 
the independent variable(s); (b) maturation, which refers to the processes within the 
participants (psychological and/or biological) taking place as a function of the passage of 
time, not attributable to the independent variable(s); (c) testing, which is sensitization to 
the posttest as a result of having completed the pretest; (d) instrumentation, which refers 
to deterioration or changes in the accuracy of instruments, devices or observers used to 
measure the dependent (outcome) variable; (e) statistical regression, which operates when 
groups are selected on the basis of their extreme scores, because these anomalous scores 
tend to regress toward the mean on repeated testing; (f) selection, which refers to the 
factors involved in placing certain participants in certain groups (e.g., treatment versus 
control), based on preferences; (g) mortality, which refers to the loss of participants and 
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their data due to various reasons, e.g., death or sickness; and (h) interactions of previous 
threats with selection. For example, a selection-maturation interaction results when the 
experimental groups are maturing at different rates based on the selection of the 
participants (Campbell & Stanley, 1963). In later writings. Cook and Campbell (1979) 
identify an additional threat to internal validity. This is ambiguity about the direction of 
casual influence when all other plausible third-variable explanations have been ruled out of 
the A-B relationship, but it remains unclear as to whether A causes B, or B causes A. 
External Validity 

This construct asks the question of generalizability. Which populations, settings, 
treatment variables and measurement variables can these results be generalized to? 
Generalizing across persons requires research samples to be representative of the 
population of interest. Generalizing across times and settings usually necessitates 
systematically administering the experimental procedure at different times and different 
settings (Parker, 1993). The inability to obtain samples that are representative of the 
populations from which they came, especially if studied in various settings, and at different 
times, results in the inability to generalize beyond the persons, time, and setting of the 
original study. Tests that do meet the representativeness criteria are, in essence, tests of 
statistical interaction. For example, if there is an interaction between a therapeutic 
treatment and ethnicity, then it can not be decisively stated that the treatment holds true 
across different ethnicities. When effects of differing magnitude exist, the researcher must 
delineate when and where the effect holds, and when and where it does not (Cook & 
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The statistical interaction threats to external validity outlined by Cook and 
Campbell (1979) are as follows: Interaction of selection and treatment (as in the previous 
example dealing with ethnicity); and interaction of setting and treatment (e.g., can a casual 
relationship obtained on a military installation also be obtained on a university campus?). 
The last interaction is between history and treatment. In this case, the question involoves 
to which period of the past or future can the results obtained be generalized. For example, 
the majority of experiments take place on university campuses, with undergraduate 
university students as participants. If an experiment was conducted on the day after a 
football loss to this university’s arch rival, then the results may not generalize even to a 
week after the loss, much less beyond the participants and setting represented in the 
original study. 

Parker (1993) reviewed and synthesized the Campbell and Stanley (1963) and the 
Cook and Campbell (1979) work and explicated two additional threats to external validity: 
The interaction of treatments with treatments, which refers to the administration of 
multiple treatments administered to the same participants, e.g., time-series designs wherein 
the effects may be cumulative; and the interaction of testing with treatment, not to be 
confused with the internal validity threat of testing when the pretest sensitizes the 
participant to the posttest. In the external validity case, the pretest may increase or 
decrease the participants responsiveness or sensitivity to the treatment. 

The above description of the most common threats to internal and external validity 
lays the groundwork for the planning of research projects. With these potential pitfalls in 
mind, the researcher is now ready to begin to plan which treatment design will be 
implemented (Lysynchuk, Pressley, d’Ailly, Smith, & Cake, 1989). The following 
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explanation of the different t)T 3 es of treatment designs and the inherent threats to their 
validity will use an “X” to represent the exposure of a group to an experimental treatment 
or event. An “O” will signify some type of observation or measurement. The Xs and Os 
in the same row will refer to the same group, and the order of the characters from left to 
right will designate the temporal order of the events. “R” will exemplify random 
assignment, if necessary (Campbell & Stanley, 1963). 

Three Pre-Experimental Designs 

In a review of the designs of the process and outcome studies published in the 
Journal of Counseling Psychology (JCP) between the years of 1964 through 1968, Kelley, 
Smits, Leventhal, and Rhodes (1970) found that 54% of the studies utilized a 
preexperimental design. Preexperimental designs are those in which there is no control 
group and/or have comparison groups that are formed nonrandomly, therefore yielding 
results which are difficult to interpret (Huck & Cormier, 1996). The three 
preexperimental designs presented by Campbell and Stanley (1963) are the one-shot case 
study, the one-group pretest-posttest design, and the static group comparison. We will 
examine these designs in the order given. 

The one-shot case study 

Much past research applied a design in which a single group was studied only once 
after a treatment was applied. These studies are diagrammed as follows; 

X O 

According to Kelley Smits, Leventhal, and Rhodes (1970) the preponderance of designs 
they reviewed in the JCP were one-shot case studies (31%). Campbell and Stanley (1963) 
refer to these studies as having “...such a total absence of control as to be of almost no 
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scientific value” (p. 5). They go on to state that “securing scientific evidence involves 
making at least one comparison” (p. 6) ...and that “It seems well-nigh unethical at least at 
the present time to allow, as theses or dissertations in education, case studies of this 
nature” (p. 7). As these studies are practically unused today, we will examine the threats 
inherent in the one-shot case design below, when they are associated with other more 
commonly used designs. 

The one-group pretest-posttest design 

This design is judged to be better than design one (Campbell & Stanley, 1976) and 
is a catalyst for understanding how many of the extraneous variables that threaten internal 
validity play out. The one-group pretest-posttest design can be reviewed by referencing 
Jemmett and Jemmett (1992), and is diagrammed as follows: 

Oi X O2 

In this design, history is one of the uncontrolled rival hypotheses, as the changes between 
Oi and 02 may have been due to events that possibly occured in addition to the 
experimenter’s X. The longer the time that elapses between the two observations, and the 
more participants for which specific events happen collectively, then the more plausible 
history becomes as a rival hypothesis (Campbell & Stanley, 1963). 

Other rival hypotheses include the participants maturing (physically or 
psychologically) between the pre and posttests, or possibly the participants do better on 
the posttest as a result of taking the pretest (testing). Maybe the measuring instrument 
changed over the course of the study (instrumentation), or certain participants may have 
selectively dropped out of the study (mortality/selection). If the participants scored 
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atypically on the pretest, they may have regressed toward the mean naturally (statistical 
regression), without any influence of X (Huck & Cormier, 1996). 

The static-group comparison 

The third preexperimental design is the static-group comparison (e.g.. Laser, 
1984). In this design, a posttest is administered to two groups, one having been 
administered the X, and the other not (a control group). When diagrammed, this design 
appears as follows: 



The basic problem with this design is the unknown status of the two groups prior to the 
administration of X, since the participants are not randomly assigned to the two groups. If 
a difference is obtained at posttest, these results may have been the influence of X. 
Alternatively though, the difference could have been an initial difference between the two 
groups. Since the participants either self-select themselves for participation into either 
group, or two existing groups are used, this is a selection threat to internal validity. 
Another threat to internal validity in this design that might threaten, even if the groups 
began as equal, is the selective drop-out rate of the participants in one group (mortality) 
(Campbell & Stanley, 1963; Huck & Cormier, 1996). 



True experimental designs yield results that are more trustworthy than the 
preexperimental designs due to the fact that random assignment is utilized, therefore 
reducing the amount of potential threats to internal validity (Huck & Cormier, 1996). 



X O 



O 



Three True Experimental Designs 
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Pretest-Dosttest control group design 

For an example of this design in practice, see Hains and Szyjakowski (1990) or 
Kush and Cochran (1993). The design is diagrammed as follows: 

R Oi X O 2 
R O3 O4 

Random assignment is employed to both groups, and both are given a pretest. One group 
is administered the X, and the other is not. A comparison of O 2 and O4 should elucidate 
any effect of the X. The unique strength of this design is the addition of the pretest, 
though there are controversies surrounding the use of pretests after random assignment 
(Heppner, Kivligham, & Wampold, 1992). In this design, many of the previous internal 
threats to validity discussed so far are accounted for. The differences attributed to history 
(and hopefully instrumentation) between Oi and O 2 would be similar to the differences 
between O 3 and O4. The threat of maturation and testing should be equally manifested 
between the experimental and the control group, statistical regression, mortality, and 
selection interaction threats are protected by the random assignment of participants 
(Campbell & Stanley, 1963), occurring probably equally across the two groups. 

Ironically, the major weakness of this design is in fact, its major strength, but for 
external validity reasons. The pretest would sensitize both the control group and the 
experimental group to the posttest in a like manner, therefore presenting no internal threat 
to validity. However, generalizing the results of a treatment that included a pretest in the 
design, to a different sample without a pretest, may yield much different results (Heppner 
et al., 1992). 
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The posttest-onlv control group design 

Here, randomization is utilized to ensure the equalization of the two groups, 
without a pretest, as in the Van Noord and Kagan (1976) study. The design is depicted in 
this way: 

R X Oi 
R O2 

Again, random assignment is employed to both groups. The X is administered to the 
experimental group, and the second group acts as a control. The internal validity of this 
design is basically solid. According to Cook and Campbell (1979), the posttest-only 
control group design is the prototypical experimental design, and most closely exemplifies 
a condition in which a casual relationship can be discerned between an independent and 
dependent variable. 

The main weakness of this design concerns external validity, i.e., the interaction of 
selection and treatment (Campbell & Stanley, 1963). Because of random assignment, the 
selection of subjects is not supposed to present a threat to internal validity. Nonetheless, it 
is often unknown whether the results of the study would generalize to another population 
(Heppner et al., 1992). For example, there could potentially be great differences between 
the results of a course on speed reading taught to a graduate class, versus a speed reading 
course taught to a high school class. Another problem deals with the absence of a pretest 
employed to reduce variability in the dependent variable. Random assignment is thought 
by some to account for this preexisting variability. But according to Huck and Cormier 
(1996) random assignment is not always random because (a) many researchers have a very 
loose definition of what randomization is, and (b) true randomization carries with it very 
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stringent criteria and many researchers are unaware of the necessary precision and falsely 
believe they have true randomization, when they do not. The suggestion given by Huck 
and Cormier is that researchers explain definitively just how they accomplished 
randomization. 

The Solomon four-group design 

When a pretest is desired, but there is concern over the effects of using a pretest, 
as in the Bateman, Sakano, and Fujita (1992) study, this design is used, notationally 
described as; 



R 


0 , 


X 


O2 


R 


O3 




O4 


R 




X 


O5 


R 






06 



This design is a combination of the pretest-posttest control group design (the first two 
groups), and the posttest-only control group (the last two groups). The main purpose of 
this design is to account for potential effects of the pretest on the posttest, and lends some 
degree of future replicability. For example, the Solomon four-group design accounts for 
the problem that the pretest-posttest control group design has, by comparing 02 to Oj to 
account for pretest sensitization, the only difference being that O 2 receives a pretest prior 
to treatment. With regard to generalizability, the researcher can compare 02 to O 4 and O 3 
to Oe. If treatment effects are found in both cases, the results will be considered strong, 
and suggest future replicability, as one replication is confirmed with the data in hand 
(Heppner et al., 1992). The major drawback of this design is the amount of time, energy, 
and resources necessary to complete the study. 
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Three Quasi-Experimental Designs 

When a true experimental design is not available to a researcher for various 
reasons, e.g., in clinical settings where intact groups are already formed, when treatment 
can not be withheld from a group, or when no appropriate control or comparison groups 
are available, the researcher can use a quasi-experimental design. As in the case of the 
true experimental design, quasi-experiments involve the manipulation of one or more 
independent variables and the measurement of a dependent variable. The major difference 
between true and quasi-experimental designs is the random assignment of participants 
(Heppner et al., 1992). Therefore, the internal validity of the quasi-experimental design is 
higher than that of the pre-experimental design, but lower than the true experimental 
design (Huck & Cormier, 1996). There are three major categories of quasi-experimental 
design: the nonequivalent-groups designs, cohort designs, and time-series designs (Cook 
& Campbell, 1979). An example of each category will be given, though the reader should 
be aware that there are many variations of each of the following examples. 

The nonequivalent-groups design 

The nonequivalent-groups design is the most frequently used quasi-experimental 
design (Heppner et al., 1992; Huck & Cormier, 1996). This design is similar to the 
pretest-posttest control group experimental design considered earlier. The difference is 
the nonrandom assignment of subjects to their respective groups in the quasi-experimental 
design. The design is diagrammed as follows, and can be further perused by referencing 
Braaten (1989): 




Non R 



Oi X O2 
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Non R O 3 O 4 

This design is one of most widely used designs in the social sciences because it is often 
interpretable. Cook and Campbell (1979) recommend this design when nothing better is 
available. The nonequivalent-groups design accounts for many of the threats to internal 
validity, except for four. The first uncontrolled threat is that of selection-maturation. As 
stated earlier, many researchers falsely believe that the administration of a pretest remedies 
the nonrandom assignment of participants, and use ANCOVA to “level” the groups. As 
has been succinctly pointed out by Loftin and Madison (1991), applying an ANCOVA 
does not always make groups equal. Furthermore, using a no-difference null hypothesis 
based on pretest scores is faulty logic as any fail-to-reject decision when testing any Ho 
does not justify believing that the null hypothesis is true (Huck & Cormier, 1996). 

Other uncontrolled threats to validity include instrumentation, differential statistical 
regression, and the interaction of selection and history (Cook & Campbell, 1979). These 
threats have been described earlier and thus warrant only mention here. 

Cohort design 

The second class of quasi-experimental designs is the cohort designs. Cohort 
designs are typically stronger than nonequivalent-groups design because cohorts are more 
likely to be closer to equal at the outset of the experiment (Heppner et al., 1992). An 
example of a cohort in this context would be, TAMU freshman in 1995 versus TAMU 
freshman in 1996. For an example from the counseling literature, see Hogg and 
Deffenbacher (1988). The basic cohort design is diagrammed as follows: 

Oi 
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X O 2 

In this design, the Oi represents a posttest administered to one cohort, while O 2 represents 
a posttest administered to the following cohort. Even though the posttests occur at 
different points in time, the posttests do occur at the same point in the progression of the 
cohort (Cook & Campbell, 1979). 

The most obvious problems with this design deal with the passage of time between 
the two cohorts, and the nonrandom assignment of participants to the cohort. The 
differences within the cohort before the treatment can be confounding. The specific 
threats to internal validity include history, when a researcher has no control over what 
events might occur in one cohort versus the other; changes surrounding instrumentation, 
testing, selection, and many interactions with selection. 

The reason why cohort designs are useful include the “quasi-comparability” (Cook 
& Campbell, 1979, p. 127) that can often be assumed between cohorts who do receive a 
treatment, and those who do not. Many times these cohort groups are more similar to 
each other than are experimental groups, especially with regard to certain demographics 
(Cook & Campbell, 1979). 

Time series desien 

The third class of quasi-experimental designs is the time-series design. These 
designs are characterized by multiple observations over time (e.g., Kivligham & Jauquet, 
1990) and involve the same participant observations to record differences attributed to 
some treatment, or similar but different participants. In the interrupted time-series design 
(the most basic of this class) a treatment is introduced at some point in the series of 
observations (Heppner et al., 1992), and this design is diagrammed as follows; 
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Oi O2 O3 O4 X O5 Oe O7 Os 
The impetus for performing this design is to observe over time, any difference after the 
treatment is implemented to discern if there is a continuous effect versus a discontinuous 
effect. A continuous effect would be treatment effects that remain stable after the initial 
discontinuity produced by the intervention. A discontinuous effect would be a result that 
decays over time. This design also accounts for effects that are instantaneous, versus 
delayed in their manifestations (Cook & Campbell, 1979), i.e., with repeated observations 
after the treatment is implemented, the researcher can ascertain how quickly the effects are 
initiated. 

The major threat to internal validity in this design is history, i.e., that variables 
other than the treatment under investigation came into play immediately after introduction 
of the treatment, e.g., a seasonal effect (the weather in a study on absenteeism from work) 
(Heppner et al., 1992). Also, sometimes instrumentation changes with the passage of 
time, as does selection (or attrition). Selection is a plausible threat when there is a 
differential attrition rate, after introduction of the treatment (Cook & Campbell, 1979). 

Summary and Conclusions 

In counseling research, which is often applied, the utility of the findings are greatly 
reduced if the findings may be attributable to something other than the treatment under 
observation. We have examined the major threats to validity and their resultant effects in 
the context of using different research designs. Every experiment is imperfect from the 
standpoint of the final interpretation, and the attempt to “fit” the results into a developing 
science (Campbell & Stanley, 1963). The previous discussion of validity and designs was 
meant to guide researchers to use better designs when developing their studies, to increase 
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awareness of the residual imperfections in their particular design to help account for 
alternative interpretations of the results. Hopefully, the call for “better and improved” 
science has been supported through this paper, and this reading will entice other students 
and researchers to contemplate long and hard before settling on a research design. 
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